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ABSTRACT 

Modem technologies radically simplify the availability of the latest scientific literature and offer new possibilities for 
sharing knowledge. Yet, most higher education institutions still rely on traditional face-to-face teaching and use e-courses 
‘only’ to supplement it. Such a combination of teaching methods is known as blended learning and is also used at the 
Faculty of Administration (FA), a member of the University of Ljubljana. The paper presents the results of a survey 
among FA students. The study’s aim was to identify which aspects of blended learning increase students’ knowledge 
level. Students evaluated 23 different aspects of blended learning on a 7-level scale. As a measure of the level of 
students’ knowledge gained from each course, we used their final grades. Applying principal component analysis, we 
extracted six dimensions of blended learning which represented predictor components in a multiple linear regression with 
final grade as the dependent variable. Since courses at FA vary in many aspects, we performed the regression analysis for 
each obligatory course individually. Our analysis revealed for which courses the final grade can be reliably predicted 
from the aspects of blended learning. The study also showed that the aspects of traditional face-to-face learning are more 
strongly linked to better grades than the aspects of e-courses. However, for some courses certain characteristics of 
e-courses play a significant role in the final grade received. 
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1. INTRODUCTION 

The Bologna Process required European universities to adapt their programmes as part of building the Higher 
European Education Area (EHEA) with aspects of harmonization, compatibility and comparability, such as 
two-cycle degree study (3 year: bachelor; 2 year: master), a comparable system of measuring student 
achievement (ECTS, competencies), student and teacher mobility, and the teaching-research relationship. 
Universities and their teachers were forced to make major changes to the pedagogical process concerning 
how to design the programmes, and the methods deployed to comply with the new requirements. A major 
novelty was the shift from a teacher-based learning style to the student-based approach whereby students 
become active participants in their own learning process, interact with the learning context and are committed 
to the learning process through which they acquire new responsibilities. Thus, the student-based style 
emphasizes the individual student and their interests, abilities, and learning styles. The teacher becomes a 
consultant of learning for individuals, helping and supporting students in achieving and building knowledge 
(Alducin-Ochoa & Vazquez-Martinez, 2016). 

Technological developments, especially in the field of information and communication, have enabled 
many innovative approaches in educational environments by introducing technology to the pedagogical 
process and by enriching the learning experience. Education via the Internet using computers or mobile 
devices using new multimedia technology is nowadays known as e-learning. E-learning is unlike the 
prevailing forms of teaching in that it is unlimited by time or space. Learning management systems (LMSs) 
provide supportive services not only for effective and efficient learning, but also to manage, guidance and 
control the e-learning process. An LMS must not only be a system that supports the sending of messages, 
keeping an online gradebook or providing handouts and learning material, but should also allow one to be an 
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active participant in e-learning, both teachers and learners, e.g. by problem-solving teamwork, question and 
answer sessions or online simulations (Campanella et al. 2008; Kim & Lee, 2008). 

Around the start of the new millennium, blended learning emerged as a new trend in teaching models and 
learning styles (Vo et al., 2017). Initially, blended learning was defined as ‘the mixture of e-learning and 
classroom learning’ (Masie, 2006) by the training field, as a promising alternative to e-learning because of 
the limitations in fostering ‘interaction, context, and remediation’ (Masie, 2006) of the latter. Subsequently, 
Graham (2006) elaborates blended learning as a combination of face-to-face instruction and 
computer-mediated instruction. Bernard et al. (2014) recommend that the proportion of blended course 
content delivered online range between 30 and 79 percent. The lower end of the range is sufficient to 
eliminate studies ‘of incidental uses of Internet, such as downloading references and turning in assignments’ 
(Means et al., 2013) and to differentiate blended learning from pure online learning (Allen & Seaman, 2009). 

Graham (2006) posits that, as an interweaving of traditional face-to-face instruction and online learning, 
blended learning allows for more interactive and reflective knowledge construction. Multi-format resources, 
archived discussions, teachers’ changing role as facilitators, and more time for discussion and reflection in 
this learning mode have been augmented by technologies (Mebane et al., 2008). 

We are encountering the rise of e-learning in the higher education area. Allen and Seaman (2013) 
observed that e-learning courses in colleges and universities in the USA are showing greater growth in their 
number than traditional courses. In a survey conducted among European higher institutions, Gaebel et al. 
(2014) reported that almost all higher education institutions selected in the study had started to embrace 
e-learning, with most using blended learning (91%) and 82% offering online learning courses. They also 
found out that three-quarters of the institutions recognize that e-learning can change the approach to learning 
and teaching methods. 

The recent boom in the blended learning industry has triggered an increase in the number of studies on 
blended learning. Considered as the ‘new normal’ mode of training (Norberg et al., 2011), the effect of 
blended learning on student performance has been researched in different contexts, e.g. in higher education, 
adult education, workplace training. There is a considerable volume of research on the benefits of satisfaction 
in higher education, especially linked to student performance (Martinez-Caro, 2011), retention (DeShields et 
al., 2005), class attendance or student engagement (Coates, 2005). Wu et al. (2012) observe that many 
e-learning studies conduct surveys to gauge learners ’ satisfaction with various items of blended learning, and 
a variety of methods is applied to assess the impact of these items on overall learning performance to provide 
a reference for improvements. Hung and Zhang (2012) point out that many researchers concentrate on 
evaluating the effectiveness of blended learning. Several models are available to measure student satisfaction, 
each with its own advantages and disadvantages (Chen, 2009). 

In many studies, the results have shown blended learning has a positive impact on student performance 
(Larson & Sung, 2009; Lopez-Perez et al., 2011; Umek et al., 2015). On the other side. Brown and Liedholm 
(2002) compared three modes of instruction and discovered that face-to-face students did significantly better 
than online students and better than blended learning students for the most complex material. Results of a 
study by Kwak et al. (2014) strongly suggest that blended learning has no impact on student performance 
whatsoever. Moreover, student performance is not affected by the introduction of blended learning regardless 
of students’ age, nationality, primary language or achievement level. But they found that introducing blended 
learning had a negative impact on male students but a positive one for female students. 

The Faculty of Administration (FA) implemented blended learning in the 2010/11 academic year, using 
LMS Moodle. It was introduced progressively over three years, with some modifications, which have proved 
to be meaningful (for more, see Umek et al., 2015). Currently, 80% of every obligatory course is held in the 
traditional way while for the remaining 20% students gain knowledge from their activities in online courses. 
Each subject has its own e-course where an e-lecture is supported by e-content followed by a quiz in order to 
check understanding of the prepared content and the three extensive assignments intended for the tutorial are 
prepared during the semester. The teachers are obliged to give feedback on the correctness of the solutions in 
those assignments 

In the study presented in this paper, we analysed 23 factors which we assumed influence students’ 
perceptions of blended learning. They relate to the characteristics of an e-course (goals, materials, and 
assignments), lecturers’ activities (assessments, responses), students’ preferences regarding learning online or 
learning in the classroom, face-to-face learning and technical details about Moodle LMS (stability, 
administrative support). In the empirical study, we performed principal component analysis in order to 
extract components that describe dimensions of blended learning. We then used these components to predict 
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the students’ grades. Since courses at the FA vary in many aspects (teachers, content, required computer 
skills, etc.), we analysed these impacts on each course separately. We believe the study presented below 
contributes some important findings to both theory and practice in the field of blended learning. 


2. EMPIRICAL RESEARCH 


2.1 Methodology and Data 

In our study, we investigated the impact of different aspects of blended learning on students’ performance. 
For this reason, we used a questionnaire based on our own recent survey (Aristovnik et ah, 2016), where 
students’ attitudes to e-learning were measured (Table 1, statements EC1-EC6, GI1-GI7), extended with 
three questions regarding face-to-face learning (Table 1, statements FF1-FF3) and seven questions on 
general attitudes to e-learning (Table 1, statements GE1-GE7), meaning that we examined a total of 23 
aspects of blended learning. The students expressed their level of agreement with the statements on an 
ordinal scale from 1 (“totally disagree”) to 7 (“totally agree”). 

Table 1. Aspects of Blended Learning 

Abb. Aspect of blended learning 

GE1 Working with computers for study purposes suits me. 

GE2 The Moodle e-learning system is easy to use. 

GE3 The Moodle system is reliable and stable (it does not crash, submitted tasks are not lost). 

GE4 I am satisfied with the support and assistance in the event of technical problems. 

GE5 Working with computers for study purposes is not difficult for me, 

GE6 E-learning contributes to higher student academic performance. 

GE7 E-learning is a quality replacement for traditional learning in the classroom. 

FF1 The content of the course interests me. 

FF2 Course lectures are interesting for me and I like to attend them. 

FF3 I find the face-to-face tutorial attractive and useful. 

EC1 The virtual classroom of the course is organized transparently. 

EC2 The goals (workload demands, grading) of this e-course were clearly stated at the start of the semester. 

EC3 This e-course offers a variety of ways of assessing my learning (quizzes, written work, forums, files...). 

EC4 I receive the teacher's comment/feedback on an assignment within less than 7 days. 

EC5 I prefer fewer lectures in the traditional way (face-to-face) and more learning material processed in the 
e-course. 

EC6 More course exercises could be carried out in the e-course instead of in the classroom. 

Gil The general impression of the e-course is good. 

GI2 Study material and tasks of the e-course are presented in a clear and understandable way. 

GI3 Finding certain activities in the e-course is simple. 

GI4 The prepared learning material and tasks are consistent with the lectures in the classroom and supplement them. 

GI5 The prepared material and assignments supplement the tutorial in the classroom. 

GI6 Learning materials and activities in the e-course helped me to effectively study this subject matter. 

GI7 The teacher gives me feedback/a response on my submissions (assignment, forum posts). 


Source: Surveys, 2015 and 2016 


The questionnaire-based survey was held in two consecutive academic years (2014/15 and 2015/16) and 
was carried out online. Students voluntarily participated in the survey, without any coercion or undue 
influence. We asked them to insert their student ID number to help us link the results obtained with their 
grades. We obtained the opinions of 639 students. They evaluated 46 undergraduate obligatory courses; on 
average, each student evaluated 5.2 e-courses. We collected 3,334 evaluations of e-courses. In addition, we 
collected the students’ grades from our student information system - the exam database for all courses 
included in the survey. 
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In order to reduce the high dimensionality of our data set and make the results more comprehensive, we 
performed principal component analysis on the questionnaire data. We used the Kaiser criterion to determine 
the number of components. A varimax rotation was used to increase the interpretability of the extracted 
components. New variables (components) were determined as arithmetic means of the variables with high 
factor loadings (above 0.5). We evaluated the new components using Cronbach’s alpha. Components with 
Cronbach’s alpha above 0.7 were kept in our data set. 

Completing the described transformation of the original datasets split the dataset (consisting of extracted 
components) into 46 subsets, each corresponding to an individual course. We added a student’s final grade to 
each dataset as the dependent variable and, since participation in the survey among higher years of the study 
was poor, certain courses received very few evaluations. Therefore, we retained only 24 courses (subsets) 
that each had more than 50 evaluations for further analysis. For each course, we performed linear regression 
analysis with extracted components as independent variables and final grade as the dependent variable. In the 
paper, we report regression models with R 2 above 0.13 and present components with a significant influence 
on the final grade. 

2.2 Empirical Results 

Principal component analysis reduced our 23 aspects of blended learning to six components, which explain 
67% of total variance (TVE). We computed Cronbach’s alpha for each of them. Since components 4 and 5 
resulted in a poor Cronbach alpha, we skipped them from further analysis and therefore provided no names 
for them. We named the four other components based on the meaning of the aspects with the highest 
loadings. The factor loadings, naming of the components with the % of total variance explained (TVE) and 
Cronbach’s alpha (factor loadings above 0.5) are shown in Table 2. 


Table 2. Component loadings for aspects of blended learning 


Component 


1 

2 

3 4 

5 

6 


aspect on 

technical 

aspect on ^ 

F2F learning 

/ 

teacher's 


e-course 

aspect 

feedback 

% of TVE 

24.8 

10.8 

9.8 8.9 

7.0 

5.7 

Cronbach's alpha 

0.914 

0.721 

0.838 0.325 

0.405 

0.741 

GE1 


0.512 




GE2 


0.722 




GE3 


0.788 




GE4 


0.819 




GE5 




0.76 


GE6 


0.499 




GE7 




0.736 


FF1 



0.831 



FF2 



0.808 



FF3 



0.627 



EC1 

0.797 





EC2 

0.815 





EC 3 

0.769 





EC4 

0.536 




0.648 

EC 5 



0.904 



EC 6 



0.916 
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Gil 

0.791 

GI2 

0.763 

GI3 


GI4 

0.712 

GI5 

0.713 

GI6 

0.637 

GI7 

0.572 


Principal component analysis revealed four blended learning dimensions, namely aspects on e-course, 
technical properties and support, face-to-face (F2F) learning, and teachers’ feedback. These four latent 
variables were later used in the regression analysis as predictors (independent variables). On the whole data 
set, we failed to detect any significant relationship between these four blended learning dimensions and the 
final grade. For this reason, we further investigated such relationships for individual courses. This approach 
has several advantages over a global analysis: it takes the courses’ specific properties into account; the final 
grades within a course are more comparable than grades overall; and, most importantly - it provides a list of 
courses in which blended learning helps to achieve better grades. 

The analysis was carried out on each individual course (with more than 50 students’ evaluations) with the 
final grade as the dependent variable. Table 3 summarizes the results of the regression analysis for six 
courses where the linear regression model sufficiently explained the variability of the final grade, i.e. with R 2 
above 0.13 (Cohen, 1992). In the table, we show unstandardized regression coefficients (B) with the 
corresponding significances (Sig.), R" and number of responses (n). 

Due to personally identifiable information, we anonymized the course names as “course 1”, “course 2” 
through to “course 6”. Instead of providing each course’s name, we reveal information about the chair to 
which the course belongs, the year of study and the study programme Courses at the FA are run by the 
teaching staff from three chairs (EPSM: Chair of Economics and Public Sector Management, ALA: Chair of 
the Administrative-Legal Area and 01: Chair of Organisation and Informatics). The undergraduate study lasts 
three years - there are two undergraduate study programmes (UN: university study programme, PS: 
professional study programme). 


Table 3. Results of regression analysis on six courses. Regression coefficient is significant at the levels 0.1 - *, 0.05 - **. 

0.01 - *** 


course 

1 

2 

3 

4 

5 

6 

chair 

EPSM 

ALA 

EPSM 

EPSM 

EPSM 

EPSM 

year 

1 

1 

2 

1 

1 

2 

study 

programme 

PS 

PS 

UN 

PS 

UN 

UN 

R 2 

0.18 

0.16 

0.15 

0.14 

0.14 

0.13 

N 

97 

177 

65 

77 

139 

113 


B 

Sig. 

B 

Sig. 

B 

Sig. 

B 

Sig. 

B 

Sig. 

B 

Sig. 

intercept 

1.96 

0.144 

5.42 

0.000 

*** 

5.89 

0.000 

*** 

3.29 

0.058 

4.79 

0.000 

*** 

5.49 

0.000 

*** 

aspect on 

e-course 

0.31 

0.504 

0.33 

0.016 

** 

-0.36 

0.306 

0.94 

0.054* 

0.57 

0.008 

*** 

-0.19 

0.374 

technical 
aspect and 
support 

0.16 

0.355 

-0.05 

0.674 

-0.14 

0.447 

-0.03 

0.856 

-0.13 

0.300 

0.07 

0.612 

aspect on 

face-to-face 

learning 

0.39 

0.012 

** 

0.31 

0.002 

*** 

0.59 

0.004 

*** 

0.24 

0.202 

0.03 

0.823 

0.02 

0.838 

teacher’s 

feedback 

-0.04 

0.900 

-0.38 

0.000 

*** 

0.02 

0.921 

-0.48 

0.021 

** 

-0.02 

0.908 

0.28 

0.006 

*** 
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Table 3 reveals that almost all significant relationships between aspects of blended learning and the final 
grade are found in courses from the Chair of Economics and Public Sector Management, with one exception 
(course 2) from the Chair of the Administrative-Legal Area. It is interesting that none of the resulting courses 
belongs to the Chair of Organisation and Informatics where more computer-based skills are used in the 
teaching and learning process. Since we restricted our analysis to courses that received more than 50 
evaluations, it is no surprise that we discovered relationships only for the courses in the first two years of 
study (due to the third-year students’ unresponsiveness). 

The aspect on e-course had a significant positive influence on the final grade in three courses (courses 2, 
4, and 5) from both chairs, but only from the first year of study. The results therefore suggest that 
characteristics related to an e-course (organization, general impression etc.) are linked to the final grade only 
in the first year of study. We suspect that students in higher years of study become used to the Moodle 
environment and all their e-course obligations (quizzes, assignments, etc.). Therefore, the organization of an 
e-course in higher years of study plays a less important role than in the first year. 

On the contrary, the technical aspect and administrative support do not have a significant impact on 
students’ grade for any course. Since this component is the only one not related to a specific course, we 
computed the correlation between its values and the final grade with all courses together. Additional 
empirical findings revealed no significant correlation (r=0.007, p=0.681) for the entire data set. We can 
therefore conclude that the technical aspect and administrative support exert no influence on students’ grades 
at the levels we analysed. 

Face-to-face learning has a significant positive impact on three courses (courses with the highest R 2 , i.e. 
courses 1, 2, and 3), from both chairs, years of study and study programme. The results suggest that the 
influence of this component is strongest especially for the course from the Chair of the Administrative-Legal 
Area. This is no surprise since courses from the ALA chair focus their teaching process on traditional 
classroom discussions. The regression coefficient of the aspect on face-to-face learning for course 2 is highly 
significant (B=0.31, p=0.002). This means that by increasing students’ attitude to the content, the quality of 
lectures and tutorials by 1 point (on a 7-level scale), we would on average expect an increase in the final 
grade of 0.31 (on a scale of 1 to 10). For the other two courses (courses 1 and 3), the increase would even be 
higher - for course 3, we expect an increase in the average grade by more than 0.5. 

The last component we analysed was the teacher’s feedback. Although we discovered it had a significant 
impact on the final grade for three courses (courses 2, 4, and 6), the empirical findings are only promising for 
course 6. The regression coefficients of teacher’s feedback are negative for courses 2 and 4 (-0.38 and -0.48, 
respectively). In the future, this surprising finding will be investigated in detail. The empirical findings 
suggest that students with higher grades expected richer and more useful feedback from the teacher whereas 
the feedback was more useful for students with lower grades. 


3. CONCLUSION 

At the Faculty of Administration, we are currently in the third year of measuring students’ satisfaction with 
blended learning. Our previous studies (Aristovnik et al., 2016; Umek et al., 2015) revealed that the 
satisfaction level with e-courses in Moodle LMS is high. In the previous analyses, we also identified which 
factors influence students’ perceptions of the usefulness of e-learning. The study presented in this paper 
represents an upgrade of our previous work. We investigated which aspects of blended learning increase 
students’ performance. From previous experience, we deduced that there is no general rule and therefore 
performed an analysis for all obligatory undergraduate courses. 

We employed principal component analysis to determine four dimensions of blended learning, namely 
aspects of e-course, technical aspect and administrative support, face-to-face learning, and teacher’s 
feedback. We failed to detect an overall (global) significant relationship between aspects of blended learning 
and the final grade. Therefore, we looked for such a relationship within each individual course. We identified 
six courses where the students’ final grade is significantly linked to components of blended learning. We 
found that e-course aspects play a significant role in the first year of study, while the face-to-face approach 
still has the strongest influence. We failed to link attitudes to the technical aspect and satisfaction with 
administrative support with final grades. The most surprising finding in our study was the identification of 
two courses where the teacher’s feedback is significantly negatively linked to the students’ final grade. We 
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suspect that the teachers of these two courses did not fulfil the expectations of students with better grades 
while students with lower grades were satisfied with their feedback. 

Our study revealed in detail that connecting satisfaction with blended learning and students’ performance 
is a challenging task. We believe that the identification of courses which reflect this relationship is an 
important achievement - both for the course teachers and managers of the Faculty. On the contrary, for the 
teachers of other courses our study suggests they should focus on other pedagogical aspects such as 
motivation for study, more interactive lectures and tutorials, and the use of recent technology as 
complementary technique in the traditional classroom. This focus will increase students’ grades and therefore 
“produce” more competent graduates. 

Our study has several limitations: we did not investigate the teachers’ role and did not ask the teachers to 
express their opinions on blended learning. Another future challenge is to increase the participation rate of 
students in the 3 ld year of study. They have much greater experience with various e-courses, overcame 
technical challenges in the first year and hold different expectations regarding the e-course quality. 
Differences among years of study suggest we should extend our questionnaire with other aspects. Some of 
the new aspects will be more suitable for the first year of study, while others will be more relevant for more 
experienced students. 
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